I. Fuzzy Artmap for Probabll..rry Est1mall0n Ii. Fuzzy Artmap

An incremental, nonparametric probability estimation procedure using the fuzzy ARTMAP (adaptive resonance theory-supervised predictive mapping) neural network is introduced. In the slow-learning mode, fuzzy ARTMAP searches for patterns of data on which to build ever more accurate estimates. In max-nodes mode, the network initially learns a fixed number of categories, and weights are then adjusted gradually. and when data sets arrive incrementally. Fuzzy ARTMAP online computations achieve both accurate probability estimates and good code compression by partitioning the input space into categories. Large or small recognition categories form to generate the best output predictions, and a variable number of recognition categories may predict each output. Categories evolve through a hypothesis testing process that incrementally incorporates information about each pattern into a knowledge base. If the system encounters a region of input space that includes several small clusters of inputs from different classes, it breaks those regions into subregions and makes a probability estimate for each subregion. ARTMAP can thus make broad, efficient generalizations, but also reduces false alarms by identifying rare or exceptional cases. Methods that try to fit the data to an assumed but incorrect distribution can fail to identify these exceptions. Simulations demonstrate that the fuzzy ARTMAP probability estimation procedure is robust, performing well in problems with different types of input distributions. Two variants of this method are described, the slow-learning mode in Section ill and the max-nodes mode in Section IV. In slow-learning mode, the system grows incrementally until it achieves a good fit to the underlying probability density function. In maxnodes mode, the user specifies an upper bound on network size. After it has reached this size the network stops growing, but additional training data can still be incorporated into the existing network to improve its probability estimates. Simulations of three probability estimation problems compare performance of both modes of fuzzy ARTMAP to that of Bayesian estimation. The three tasks requiring probability estimation for a simple two-Gaussian distribution are discussed in Section V, a trimodal distribution is discussed in Section VI, and a difficult problem in which inputs to each class form distributions that are 97-modal, with modes falling on two intertwined spirals, is discussed in Section VII. Fuzzy ARTMAP provides accurate estimates in both modes for all three tasks. Finally, Section Vill includes a discussion of the ARTMAP algorithmic variations, and the Appendix illustrates slow learning with a computational example. I. FuzzY ARTMAP FOR PROBABll..rrY EsT1MAll0N M ANY pattern recognition applications require an estimate of the probability that an input belongs to a given class. In a medical database, for example, a set of measurements can be used to estimate the probability that a patient will require a long stay in the hospital. Different groups of diagnostic factors may be associated with a single outcome, .and it is possible that no single combination of variables forms a unique set of predictors. Fuzzy ARTMAP (adaptive resonance theory-supervised predictive mapping), [I], [2] discussed in Section II, is a neural network that automatically selects complex combinations of factors on which to build accurate predictions for application to problems such as medical prediction and handwritten character recognition [3]-[5]. Fuzzy ARTMAP is able to create a stable memory structure even with fast, on-line learning. With fast learning, the network would regard each on-line training point as potentially informative, possibly an important rare case, and record its prediction in the set of learned categories. In this training mode, however, noisy data can lead to category proliferation. A procedure that uses fuzzy ARTMAP slow learning for probability estimation in a noisy input environment is developed here. Unlike parametric probability estimators, fuzzy ARTMAP does not depend on a priori assumptions about the underlying data. The network can make accurate probability estimates even when the underlying distributions are unknown Manuscript received August 6, 1993; revised November 12, 1994 and March 21, 1995. This work was supported in part by the Air Force Office of Scientific Research (AFOSR F49620-92-J-O175 and AFOSR 90-0225), the Advanced Research Projects Agency (ONR NOOO14-92-J-4015), the National Science Foundation (NSF IRI 94-00530 and NSF 1RI-90-01659), and the Office of Naval Research (ONR NOOOl4-91-J-4100, ONR NOOOl4-9~, and ONR NOOOl4-95-0657). The authors are with the Center for Adaptive Systems and Department of Cognitive and Neural Systems, Boston University, Boston, MA 02215 USA. IEEE Log Number 9413442. II. FuZzY ARTMAP Fuzzy ARTMAP (Fig. 1) includes a pair of ART modules [6] (ART a and ART b) that create stable recognition categories 1045-9227/95$04.00 @ 1995 IEEE 1332 IEEE TRANSACTIONS ON NEURAL NE1WORKS. YOLo 6. NO.6. NOVEMBER 1995 on a long-tenn average of its experience, while values of {Jab near one allow adaptation to a rapidly changing environment. The Appendix includes a computational exampl~ of this slow-